Submission Tests

"Mistakes are not errors but partially correct solutions with underlying logic." - Alina Tugend

An exercise's submission correctness test (SCT) is code that course creators write to check the correctness of a learner's code submission and guide them to the right solution.

When a learner runs their solution to an exercise, your SCT has access to the variables, functions, and objects within that code at runtime. Course creators can then use these constructs to determine whether or not a learner's submission should pass or fail.

Show documentation for:

Basic Tests

Submission tests run in the same environment as a learner's submitted code. This means that when you write a function in the Submission Correctness Test editor box in the Course Creation Tool, you are automatically inside a submission test function and have access to the scope of that function.

Failing or passing a submission is determined by the return value from your test function. Return Boolean true to pass a learner's submission, and return a false value to fail a learner's submission.

For example, if you want the learner's submission to always pass, return true. 

return true;

If you return false, students will see the error message "Oops, try again." To provide a more helpful error message, return a string that will be displayed as an error. 

return "Make sure that you have closed all your parentheses.";

Back to top

Accessing Student Code

The submission correctness test provides you with access to information regarding the code the learner has written and the status of the environment in which it was run. This information can be accessed and manipulated as follows:

Code

The variable code contains the string of the entire contents of the code that the learner entered. For example, if you're expecting the submitted code to have at least one plus (+) sign, you could write the correctness test: 

return code.indexOf('+') > -1;

Result

The variable result contains the result of evaluating the students' code. It can be of any data type. If you're anticipating the result of a learner's entry to be a number, you could use the test: 

return typeof result === "number";

If you're expecting the number two you could simply use: 

return result === 2;

This test would work fine if the learner enters the expected code e.g. 1 + 2. But what if the learner enters a single number. The submission test would pass it even though it's wrong. We could further strengthen the test by adding the plus check: 

return typeof result === "number" && code.indexOf('+') > -1;

That's better and it works. However its not 100% right. You could always write RegExp but then it may not catch all cases. The correct way to do it is to use a proper code analysis tool. Helpers for this type of check will be available in our library soon!

Error

The error object contains the error thrown by a learner's code. If there is no error, it is the null object. This is useful when anticipating learner mistakes.

For example, if you wanted to test whether or not the learner's code threw a reference error, you could use: 

if (error instanceof ReferenceError) {
  return "Make sure you put quotes around your name."
}

RegExp

You can also use regular expressions to check for specific strings in learner code. This is fairly rigid as a test, difficult to get just right, and is difficult to read. We don't recommend it, but it is useful in some cases.

For example, it is currently the only way to check if tags are closed in HTML. 

if (code.html.match(/\<\/a\>/)===null) {
  return 'Did you remember to close the "a" tags?';
}
return true;

Helper Libraries

In an effort to provide as rich an experience as possible to course creators, we have in addition to providing course creators with access to the aforementioned variables provided a suite of functions to make it easier to access various aspects of the runtime environment.

Say you ask the user to define a function called foo and to call said function. You could write a submission test like this: 
return typeof foo === "function" && code.match(/foo\s*\(/) != null;

Although this is a relativaly strong check, it could go wrong in many ways and is unneccessarily complex. To simplify such a check, use the CC.calls() function to test if a function has been defined and invoked within the user's code submission. 

return typeof foo === "function" && CC.calls('foo').length > 0;

CC is our helper library's namespace (short for CodeCademy). We maintain the same API for all languages supported by our system unless mentioned otherwise.

CC.calls

A function that takes a function name and returns an array of the arguments that were passed to the function every time it was called in the code.

Student code: 

foo(1, 2);
foo('x', 'y');

Then by calling: 

CC.calls('foo');

In our submission test would return an array looking like [[1, 2], ['x', 'y']]

CC.methods

A function that takes a method name and an object and returns an array of arguments that were passed to the method called on that object.

Student code: 

var cow = {};
cow.says = function(sound) { console.log(sound); }
cow.says("Moo");

Then by calling: 

CC.methods("says", "cow");

In the SCT, we'd get an array looking like [[""Moo""]]. (Note that for arguments including quotes, the quotes are repeated.)

CC.equals

Takes any number of objects and checks the deep equality of these objects. This is equivalent to calling JSON.stringify on each item and comparing the resulting strings. 

CC.equals([1,2], [1,2]); //true

CC.prints

Returns an array of strings printed to the console.

Student Code: 

console.log('fooooo');
console.log('baaaar');
CC.prints(); // ['fooooo', 'baaaar']

CC.printed

Given a string, returns a boolean representing whether or not the string was printed to the console.

Student code: 

console.log('fooooo');
CC.printed('fooooo'); //true
CC.printed('baaaar'); //false

Back to top

Combining conditions

It should be noted that conditional tests can always be combined via the && and || operators.

For example, if you wanted to test that the user entered a number or a string, you could use: 

return typeof result === "number" || typeof result === "string";

Back to top

Helpful Error Messages

You can provide students with custom error messages from within the submission test by returning a string.

Lets go back to the addition example and further enhance it by giving the user hints about what they did wrong. 

if (typeof result !== "number")
  return "You should probably use numbers";

if (code.indexOf('+') === -1)
  return "Maybe you should use the plus sign for addition"

return true

You can also try to guess what the learner did wrong from the type of error thrown from evaluating her code.
For example if your trying to get her to type in her name and got a reference error then chances are she left out the quotes: 

if (error && error instanceof ReferenceError) {
  return "Make sure you surround your name with quotes!"
}
if (typeof result === "string") {
  return true;
}
return false;

Back to top

Debugging and Preview Mode

Preview Mode adds an in-browser deubgging console that will show helpful log and error messages. Preview Mode console will show the following:

  • Errors in the SCT.

  • In web courses, errors from the learner code.

  • When SCT is waiting on an async event and when its resolved.

Note that in Preview Mode errors in the SCT will cause the submission to fail while otherwise those same errors will pass in Student Mode so they don't block on an exercise that is impossible to pass.

Thoroughly test your submission test functions before publishing your course.

Back to top

Ruby

Basic Tests

Submission tests run in the same environment as a learner's submitted code. This means that when a student runs his or her code in an exercise, your SCT has access to all variables, functions, and objects within that code at runtime.

You can simply write SCTs to assume that those constructs exist. If the learner failed to use the necessary construct, however, your SCT will have to handle the generated exception appropriately (see Helpful Error Messages ). Failure to handle the exception will result in the generic message: Oops, try again.

Failing or passing a learner's submission is determined by the return value from your SCT. Return true to pass a learner's submission, and return a false value to fail a learner's submission.

For example, if you want the learner's submission to always pass, then your SCT can contain the following: 

return true

If you simply return false, then students will see the error message: "Oops, try again." To provide a more helpful error message, you can return a string that will be displayed as the error message. This can be achieved with the following: 

return "Make sure to use the .each method in your code."

Of course, these types of SCTs are not very helpful, since they don't really check that the user's code works. You should avoid writing SCTs that only contain the above snippets. In the next section, we'll cover how to access constructs within a learner's submitted code.

Back to top

Accessing Student Code

The SCT provides you with access to information regarding the learner's code and the status of the environment in which it was run. This information can be accessed and manipulated by utilizing the following:

Code

The code variable contains a string representation of the learner's submission. You can perform checks against this variable to determine if the user's code contained the necessary operators.

For example, if the learner were to run the code: 

my_sum = 2 + 2

Then the variable code would contain the string "mysum = 2 + 2"

If you wanted to make sure that the addition operator + was used, you could simply check that the code contains more than 0 instances of the addition operator, This can be achieved with the following: 

return code.include? "+"

Remember, code is just a string variable, so you can use Ruby's built-in string methods to determine if the student's code solves the exercise. The above code uses Ruby's #include? method to determine how many times the string "+" occurs in the student's code.

You might think to use the code variable for checking if a necessary variable was declared in the learner's code, but it might be more appropriate to assume that the variable exists and try calling it. If the learner didn't define the required variable, then your SCT will trigger a NameError exception.

See the section Helpful Error Messages for a discussion of handling exceptions with Ruby's begin/rescue/end constructs.

Result

The variable result contains the result of evaluating the students' code. If you'd like to make sure that the result of a learner's submission is an integer, you could use the following in your SCT: 

return result.is_a? Integer

If you're expecting the evaluation of the student's code to be a specific value, you could then use the following: 

return result == 4

Again, this holds for any data type, not just integers.

The problem is that if you only check that the answer is correct, you allow the learner to just simply enter the answer and hit "run." Unfortunately, your SCT will pass them. Instead, if you want ensure that an operation be used to yield the answer, you can use the code variable in conjunction with the result variable. This can be seen from the following: 

return code.include? "+" && result == 4

The above SCT ensures that the student's code produces the answer 4 and includes an addition operation. However, this is still not perfect. This SCT will pass the user given the following incorrect code: 

fooled_you = 100 + 50
4

This passes because it satisfies the fact that the result was the number 4 (from the second statement) and that the code contained at least one use of the addition operator. The correct way to do it is to use a proper code analysis tool. Helpers for this type of check will be available soon!

Exception

The exception object contains the exception thrown by a student's code. If there is no exception, its value is nil . This is useful when anticipating student mistakes.

For example, the following student code generates a SyntaxError exception: 

my_name = "Eric Weinstein

In your SCT, you can handle syntax errors with the following: 

if error.kind_of? SyntaxError
  return "Make sure to use double quotes around your name."

This shows the following output to the learner: 

Oops, try again. 
Make sure to use double quotes around your name.

This is discussed further in the section Helpful Error Messages .

Back to top

Helper Libraries

We've provided you with a suite of functions to make it easier to check various aspects of a learner's submission. These functions are contained within codecademy_lib , also known as the CC library.

For Ruby, this library is still under development. We'll incorporate additional helper methods soon!

prints

Returns an array of strings that were printed to the console after running the learner's code. You do not need to prefix prints with CC .

Student Code: 

puts 'fooooo'
puts 'baaaar'

SCT: 

prints # ['fooooo', 'baaaar']

printed

A separate printed method does not yet exist, but prints.include? can be used instead.

Given a string, prints.include? returns a boolean value representing whether or not the string was printed to the console.

Student code: 

puts 'fooooo'

SCT: 

prints.include? 'fooooo' # true
prints.include? 'baaaar' # false

Note that this will not work for the totality of the student's printed output, as prints returns an array in which each newline-delimited string is an element.

Back to top

Combining conditions

It should be noted that conditional tests can always be combined via the && and || operators.

For example, if you wanted to test that the user entered an integer or a string, you could use: 

return result.is_a? Integer || result.is_a? String

Back to top

Helpful Error Messages

It's important to understand that SCTs are meant not only to check the correctness of a student's code, but to help guide him or her to the right answer(s). Many students get discouraged when the console prints unhelpful or confusing error messages. Hence, it's your job to avoid those obstacles and provide encouraging and useful messages for likely errors.

You can provide students with custom error messages from within the submission test by returning a string.

In the simple case, if the learner's solution doesn't contain required constructs, you can return a string that explains why their solution is incorrect.

For example, let's go back to the addition example and further enhance it by giving the learner hints about what they did wrong. 

unless result.is_a? Integer
  return "Check to make sure your code results in an integer."

unless code.include? "+"
  return "It doesn't look like you're using the addition operator."

return true

Rather than using if !true in Ruby, it's often cleaner to use unless true . However, this construct should be avoided if the statement includes an else . In that case, if / else is the better construction.

Where are errors generated?

It's important to note that errors can appear both in the user's code and the SCT.

Errors in the student's code can typically be found in the exception variable, as discussed in the Accessing Student Code section. You can check for these errors using a simple conditional check and return a message that helps the student understand how they generated an error.

Errors in your SCT can result from two situations: a syntax error in your code or an exception generated when trying to use constructs defined in the user's code.

For the first type of SCT error, where your code is flawed, those errors are reported to you via the preview mode console discussed in the Debugging and Preview Mode section. These errors should be straightforward for you to remedy.

The second type of SCT error, where your SCT uses non-existent constructs in the student's code, generates exceptions that you need to handle using Ruby's begin/rescue/end syntax.

For example, if your exercise asked the user to type in his or her name and the submission generated a syntax error, it's very likely that the student missed a starting or closing quote: 

if exception.kind_of? SyntaxError
  return "Make sure you surround your name with quotes!"

unless result.is_a? String
  return "Make sure your code produces a string value."

return true

Common Exceptions

NameError

Raised when a variable is used in a context but not declared. This exception can be triggered form both the user's code and the SCT.

An example of learner code that generates the exception:

puts first_name   # first_name not declared

The SCT to catch this: 

if exception.kind_of? NameError
  return "It looks like you tried to use a variable without declaring it first."

An example of an SCT that generates the exception:

If the exercise called for a variable first_name to be defined and the student's code was: 

"Oxnard Montalvo"

The following SCT will attempt to use the expected variable first_name only to see that it doesn't exist, triggering a NameError exception within the SCT. 

first_name == 'Ryan'   # NameError

The SCT to catch this exception involves wrapping the SCT code in a begin/rescue/end block: 

begin
  first_name   # NameError
rescue NameError:
  return "Did you remember to declare a variable called first_name?"
end

# If they don't trigger an exception, let's pass them.
return true

The learner will then see the following output: 

Oops, try again. 
Did you remember to declare a variable called first_name?

SyntaxError

Raised when the student generates any kind of syntax error. This exception should be caught using the exception variable, and error messages should be context-dependent. For instance, if the student is learning how to use if/else statements and his or her code raises a syntax error, your SCT might be: 

if exception.kind_of? SyntaxError
  return "It looks like your if/else syntax isn't quite right."

ZeroDivisionError

Raised when the denominator of a division or modulo operation is zero. This exception should be caught using the exception variable.

Student Code: 

100 / 0

SCT to catch this: 

if exception.kind_of? ZeroDivisionError
  return "Dividing by zero isn't allowed!"

For more information on exception handling and a list of exceptions, please see the Ruby Exception Class documentation.

Back to top

Debugging and Preview Mode

Preview Mode adds an in-browser debugging console that will show helpful log and error messages. This mode is activated when you hit Save and then Preview when editing an exercise. You'll then notice a console window at the bottom of the page.

The Preview Mode console will show you the following:

  • Errors in your SCT generated from syntax and other creator-caused errors.

  • When the SCT is waiting on an asynchronous event and when it's resolved.

When a student attempts to solve an exercise that has a flawed SCT, the student shouldn't be blocked from progressing through the course. To this end, we've instated logic such that broken SCTs will automatically pass the student.

Please thoroughly test all submission correction tests before publishing your course!

Back to top

Basic Tests

Submission tests run in the same environment as a learner's submitted code. This means that when a learner runs their code in an exercise, your SCT has access to the variables, functions, and objects within that code at runtime.

You can simply write SCTs to assume that those constructs exist. If the learner failed to use the necessary construct, however, your SCT will have to handle the generated exception appropriately (see Helpful Error Messages ). Failure to handle the exception will result in the generic message: Oops, try again.

Failing or passing a learner's submission is determined by the return value from your SCT. Return True to pass a learner's submission, and return a False value to fail a learner's submission.

For example, if you want the learner's submission to always pass, then your SCT can contain the following: 

return True

If you simply return false, then students will see the error message: "Oops, try again." To provide a more helpful error message, you can return a string that will be displayed as the error message. This can be achieved with the following: 

return "Please check that you have closed all of your parentheses."

Of course, these types of SCTs are not very helpful, since they don't really check that the user's code works. You should avoid writing SCTs that only contain the above snippets. In the next section, we'll cover how to access constructs within a learner's submitted code.

Back to top

Accessing Student Code

The submission correctness test provides you with access to information regarding the learner's code and the status of the environment in which it was run. This information can be accessed and manipulated by utilizing the following in your SCTs:

Code

The code variable contains a string representation of the learner's submission. You can perform checks against this variable to determine if the user's code contained the necessary operators.

For example, if the learner ran the code: 

my_sum = 2 + 2

Then the variable code would contain the string "mysum = 2 + 2"

If you wanted to make sure that the addition operator + was used, they could simply check that the code contains more than 0 instances of the addition operator. i.e., The code should contain at least one instance of the addition operator. This can be achieved with the following: 

return code.count("+") > 0

Remember, code is just a string variable, so you can use Python's built-in string methods to determine if the learner's code solves the exercise. The above code uses Python's count() function to determine how many times the string "+" occurs in the learner's code.

You might think to use the code variable for checking if a necessary variable was declared in the learner's code, but it might be more appropriate to assume that the variable exists and try calling it. If the learner didn't define the required variable, then your SCT will trigger a NameError exception.

See the section Helpful Error Messages for a discussion of handling exceptions with Python's try/except constructs.

Result

The variable result contains the result of evaluating the students' code. It can be of any data type. If you'd like to make sure that the result of a learner's submission is an integer, you could use the following in your SCT: 

return type(result) == int

If you're expecting the evaluation of the learner's code to be a specific value, you could then use the following: 

return result == 2

Again, this holds for any data type, not just integers.

The problem is that if you only check that the answer is correct, you allow the learner to just simply enter the answer and hit run. Unfortunately, your SCT will pass them. Instead, if you want ensure that an operation be used to yield the answer, you can use the code variable in conjunction with the result variable. This can be seen from the following: 

return result == 2 and code.count('+') > 0

The above SCT ensures that the learner's code produces the answer 2 and has an addition operation. However, this is still not perfect. This SCT will pass the user given the following incorrect code: 

fooled_you = 100 + 50
2

This passes because it satisfies the fact that the result was the number 2 (from the second statement) and that the code contained at least one use of the addition operator. The correct way to do it is to use a proper code analysis tool. Helpers for this type of check will be available soon!

Error

The error object contains the exception thrown by a learner's code. If there is no error, its value is None . This is useful when anticipating learner mistakes.

For example, the following learner's code generates a SyntaxError exception: 

name = "Joel Kemp

In your SCT, you can handle syntax errors with the following: 

if type(error) == SyntaxError:
  return "Make sure that you have double quotes around your name."

This shows the following output to the learner: 

Oops, try again. 
Make sure that you have double quotes around your name.

This is discussed further in the section Helpful Error Messages .

Back to top

Helper Libraries

We've provided you with a suite of functions to make it easier to check various aspects of a learner's submission. These functions are contained within codecademy_lib , also known as the CC library.

For Python, this library is still in development and we will incorporate additional helper methods soon.

prints

Returns an array of strings that were printed to the console after running the learner's code.

Student Code: 

print 'fooooo'
print 'baaaar'

SCT: 

CC.prints() # ['fooooo', 'baaaar']

printed

Given a string, the printed() function returns a boolean value representing whether or not the string was printed to the console.

Student code: 

print 'fooooo'

SCT: 

CC.printed('fooooo') # True
CC.printed('baaaar') # False

Back to top

Combining conditions

It should be noted that conditional tests can always be combined via the and and or operators.

For example, if you wanted to test that the user entered an integer or a string, you could use: 

return type(result) == int or type(result) == str

Back to top

Helpful Error Messages

It's important to understand that SCTs are meant to guide students to the right answers, in addition to checking the correctness of their code. Many students get discouraged when the console prints uninformative errors. Hence, it's your job to avoid those obstacles and provide encouraging messages for likely errors.

You can provide students with custom error messages from within the submission test by returning a string.

In the simple case, if the learner's solution doesn't contain required constructs, you can return a string that explains why their solution is incorrect.

For example, let's go back to the addition example and further enhance it by giving the learner hints about what they did wrong. 

if type(result) != int:
  return "Check to make sure that you're using numbers."

if code.count('+') == 0:
  return "It doesn't seem like you're using the addition operator."

return True

Where do errors get generated?

It's important to note that errors can be generated from both the user's code and the SCT.

Errors in the learner's code can typically be found in the error variable, as discussed in the Accessing Student Code section. You can check for these errors using a simple conditional check and return a message that helps the learner understand how they generated an error.

Errors in your SCT can result from two situations: a syntax error in your code or an exception generated when trying to use constructs defined in the learner's code.

For the first type of SCT error, where your code is flawed, those errors are reported to you via the preview mode console discussed in the Debugging and Preview Mode section. These errors should be straightforward for you to remedy.

The second type of SCT error, where your SCT uses non-existent constructs in the learner's code, generates exceptions that you need to handle using Python's try/except syntax.

For example, if your exercise asked the learner to type in their name and their submission generated a syntax error, then it's very likely that the learner missed a starting or closing quote: 

if error and type(error) == SyntaxError:
  return "Make sure you surround your name with quotes!"

if type(result) == str:
  return True

return False

Likely Triggered Exceptions

NameError

Raised when a variable or function is used in a context but not declared/defined. This exception can be triggered form both the learner's code and the SCT.

An example of learner code that generates the exception:

name = first_name   #first_name not declared

The SCT to catch this: 

if type(error) == NameError:
  return "You tried using a variable that doesn't exist."

An example of an SCT that generates the exception:

If the exercise called for a variable first_name to be defined and the learner's code was: 

"Bob"

The following SCT will attempt to use the expected variable first_name only to see that it doesn't exist -- triggering a NameError exception within the SCT. 

print "Welcome", first_name   # NameError

The SCT to catch this exception involves wrapping the SCT code in a try/except block: 

try:
  print "Welcome", first_name   # NameError
except NameError:
  return "Make sure that you've defined the variable first_name."

# If they don't trigger an exception, then let's pass them.
return True

The learner will then see the following output: 

Oops, try again. 
Make sure that you've defined the variable first_name.

ZeroDivisionError

Raised when the denominator of a division or modulo operation is zero. This exception should be caught using the error variable.

Student Code: 

2 / 0

SCT to catch this: 

if type(error) == ZeroDivisionError:
  return "Dividing by zero is not allowed."

For more information on Exception handling and a list of Exceptions, please see the Python Error Handling documentation.

Back to top

Debugging and Preview Mode

Preview Mode adds an in-browser debugging console that will show helpful log and error messages. This mode is activated when you hit Save and then Preview when editing an exercise. You'll then notice a console window at the bottom of the page.

The Preview Mode console will show you the following:

  • Errors in your SCT generated from syntax and other creator-caused errors.

  • When the SCT is waiting on an async event and when its resolved.

When a learner attempts to solve an exercise that has a flawed SCT, the learner shouldn't be blocked from progressing through the course. To this end, we've enstated that broken SCTs will automatically pass the learner.

Please thoroughly test your submission correction tests before publishing your course.

Back to top

Basic Tests

Submission tests run in the same environment as a learner's submitted code. This means that when you write a function in the Submission Correctness Test editor box in the Course Creation Tool, you are automatically inside a submission test function and have access to the scope of that function.

Failing or passing a submission is determined by the return value from your test function. Return Boolean true to pass a learner's submission, and return a false value to fail a learner's submission.

For example, if you want the learner's submission to always pass, return true. 

return true;

If you return false, students will see the error message "Oops, try again." To provide a more helpful error message, return a string that will be displayed as an error. 

return "Make sure that you have closed all your parentheses.";

Back to top

Accessing Student Code

The submission correctness test provides you with access to information regarding the code the learner has written and the status of the environment in which it was run. This information can be accessed and manipulated as follows:

Code

The variable code contains the string of the entire contents of the code that the learner entered. For example, if you're expecting the submitted code to have at least one plus (+) sign, you could write the correctness test: 

return code['script.js'].indexOf('+') > -1;

Notice that unlike when accessing code in a JavaScript lesson we have to tell the code descriptor which file we want it to look in. Here We noted that we wanted to look in the file ['script.js'].

RegExp

You can also use regular expressions to check for specific strings in learner code. This is fairly rigid as a test, difficult to get just right, and is difficult to read. We don't recommend it, but it is useful in some cases.

For example, it is currently the only way to check if tags are closed in HTML. 

if (code['script.js'].match(/\<\/a\>/)===null) {
  return 'Did you remember to close the "a" tags?';
}
return true;

Helper Libraries

In an effort to provide as rich an experience as possible to course creators, we have provided a suite of functions to make it easier to access various aspects of the runtime environment.

$expect.js

One of the most useful libraries we have created for web courses is the $expect.js library. This library allows course creators to more easily access Document Object Model objects, and test them for various conditions. For example, if we want to see if there exists at least one element in a jQuery collection we can use:

exist

Check the existence by asserting that there is at least one element in the jQuery collection. 

$expect('div').to.exist();
$e('div').to.exist('Please add a div to the page!');

As you can see the $expect library can be called using either its full form $expect() or shorthand $e(). It uses a system of chaining language-like statements to automate the process of checking DOM elements. You will see how these can be combined and manipulated in the following examples.

Error message

As mentioned above, $expect ships with default error messages for each of the assertion API's available. But it also gives the user the ability to override these error messages.. The final argument to any $expect phrase will override the default error message if passed. The error function handles string type arguments as well as function aliases that will be evaluated and have their return value passed as the error message. The passed in function could act as a cleanup function before throwing the assertion error. It would be passed a boolean stating whether the assertion would throw or not. 

$expect('div').to.exist(function (willThrow) {
  // Some cleanup code.
  return 'Please add a div to the page.'
});

The implications of the error function expand beyond simple messages. You can rely on anything executed by $expect to be performed in a safe environment. Meaning that even if there are faulty references or other errors in the objects or code passed to $expect, it will return the appropriate error message without crashing. This can be very useful in web submission correctness tests where we are often unsure if objects were created by the learner. Now let's review some of $expects other abilities.

items / elements / length

Asserts the .length property. 

$expect('ol > li').to.have.items(4);

above / greaterThan

Asserts that the .length property is greater than a given number. 

$expect('li').to.be.above(4);

below / lessThan

Asserts that the .length property is less than a given number. 

$expect('li').to.be.lessThan(5);

be / a / an

Asserts that each element in a jQuery collection matches the passed in selector.

$expect('div').to.be('.widget');
$expect('input').to.be('[type=text]');
$expect('.win').to.be.a('div');
$expect('.list').to.be.an('ol');

Internally calls $().is so it can be passed either a selector, a function, a jQuery object, or an element.
For more info check out the jQuery docs. 

$expect('h1').to.be($headers);

eql / equal

Asserts that one jQuery collection has the exact same elements as another.
Can accept a jQuery collection or simply a selector 

$expect('li.first').to.be.equal('li:first');
$expect('div').to.be.equal($('.all-the-divs'));

attr

Asserts the existence of an attribute and its equality if a value was passed in.

$expect('.container').to.have.attr('id', 'content');
$expect('.some-input').to.have.attr('value');

text

Asserts that an element has the exact same text.
If a number is passed in then the length of the text will be checked.
If a RegExp is passed in then it will be matched against the text. 

$expect('.link-1').to.have.text(10);
$expect('.link-1').to.have.text(/code/i);
$expect('.link-1').to.have.text('Codecademy');
$expect('.link-2').to.have.text('Google', 'Why not?');

contain

Asserts that an element contains a certain text. By default punctuation, whitespace, and case would are ignored. Pass in a second true argument to ensure a strict check. 

$expect('body').to.contain('author');
$expect('.links').to.contain('people');
$expect('.content').to.contain('Amjad', 'My name must exist and be capitalized');

Dimension check

Expect can check the width, innerWidth, outerWidth, height, innerHeight,
outerHeight, scrollTop, and scrollTop of an element.
It can be passed a number or string, and those can optionally have a comparison operator prepended to them. 

$expect('.nav').to.have.width(250);
$expect('.header').to.have.innerWidth('>= 50')
        .and.to.have.innerWidth('<= 250');

value / val

Asserts that an input element has a certain value. Calls $().val. 

$expect('input.password').to.have.val('PlainText');

html

Asserts that an element has an html string. Calls $().html. 

$expect("body").to.have.html("<div>foo</div>");

Traversing

Asserts the existence of element in different directions of the DOM tree.
Relies on jQuery's traversal methods. 

$expect("body").to.have.children(".foos");
$expect("#so-lonely").to.have.siblings(".party-elements");

Importantly, $().find is aliased to $().have 

$expect('body').to.have('input');

We also have the rest of the normal tree traversal methods associated with jQuery available to us with the same syntax. Here is the full list

class

Asserts the existence of a class or multiple space separated classes on each element of a collection.

$expect('input[type=text]').to.have.class('on field');

Shorthand attributes

Convenience methods for checking the following attributes and selectors:
visible, hidden, selected, checked, disabled, empty. 

$expect('h2').to.be.hidden();
$expect('input.submit').not.to.be.hidden('Please hide the submit button for now!');
$expect('body').not.to.be.empty();

Chaining

You can chain assertions on an object just like you can chain methods in jQuery.

And

Chains assertions on the original object.

$expect('div.container').to.exist().and.not.be.empty().and.to.have.width('>= 250');

that / which

Chains assertions on different elements after calling any of the traversal methods.

$expect('ul.todos').to.exist().and.to.have.children('li.items').that.has.css('border', '1px solid red').and.has.attr('data-id');

When chaining on traveresed elements just as in jQuery you can always call .end() to get the original object back. 

$expect('div.container').to.have.siblings('.pane').that.has.css('float', 'left').end().to.be('.loading');

Testing events

You can even use $expect to test events safely and quickly. Just use the form below 

// The actual call to the async event. Expect the window to scroll.
// And when that happens expect the nav-bar to become position fixed.
$expect(window).to.scroll(function () {
    $expect('.nav-bar').to.have.css('position', 'fixed');
  });
});

CC (CodeCademy) Library

Say you ask the user to define a function called foo and to call said function. You could write a submission test like this: 

return typeof foo === "function" && code.match(/foo\s*\(/) != null;

Although this is a relativaly strong check, it could go wrong in many ways and is unneccessarily complex. To simplify such a check, use the CC.calls() function to test if a function has been defined and invoked within the user's code submission. 

return typeof foo === "function" && CC.calls('foo').length > 0;

CC is our helper library's namespace (short for CodeCademy). We maintain the same API for all languages supported by our system unless mentioned otherwise.

CC.calls

A function that takes a function name and returns an array of the arguments that were passed to the function every time it was called in the code.

Student code: 

foo(1, 2);
foo('x', 'y');

Then by calling: 

CC.calls('foo');

In our submission test would return an array looking like [[1, 2], ['x', 'y']]

CC.equals

Takes any number of objects and checks the deep equality of these objects. This is equivalent to calling JSON.stringify on each item and comparing the resulting strings. 

CC.equals([1,2], [1,2]); //true

CC.prints

Returns an array of strings printed to the console.

Student Code: 

console.log('fooooo');
console.log('baaaar');
CC.prints(); // ['fooooo', 'baaaar']

CC.printed

Given a string, returns a boolean representing whether or not the string was printed to the console.

Student code: 

console.log('fooooo');
CC.printed('fooooo'); //true
CC.printed('baaaar'); //false

Back to top

Combining conditions

It should be noted that conditional tests can always be combined via the && and || operators.

For example, if you wanted to test that the user entered a number or a string, you could use: 

return typeof result === "number" || typeof result === "string";

Back to top

Helpful Error Messages

You can provide students with custom error messages from within the submission test by returning a string.

Lets go back to the addition example and further enhance it by giving the user hints about what they did wrong. 

if (typeof result !== "number")
  return "You should probably use numbers";

if (code.indexOf('+') === -1)
  return "Maybe you should use the plus sign for addition"

return true

You can also try to guess what the learner did wrong from the type of error thrown from evaluating her code.
For example if your trying to get her to type in her name and got a reference error then chances are she left out the quotes. You have 2 common options if you have to check a variable value, and so cannot use the $expect.js library to check a DOM element. You can use a try catch statement in anticipation of an error: 

try
  {
    if (typeof sampleVariable === "string") {
      return true;
    }
catch(err)
  {
    return "sampleVariable did not exist! Make sure it is being initialized.";
  }

This method is more robust and allows you to check any operation you perform. However, if all you are checking is an if statement, then you can use this method as well: 


if(typeof sampleVariable == "undefined" || sampleVariable != 4)
  return "You did not correctly set sampleVariable to 4. Please make sure your code correctly gives it a value";

This works because if the first check in the if statement finds that sampleVariable is indeed undefined, it will skip the second check and jump directly into the if clause. This behaviour within JavaScript allows us to avoid the second check for a nonexistant variable. Back to top

Debugging and Preview Mode

Preview Mode adds an in-browser debugging console that will show helpful log and error messages. Preview Mode console will show the following:

  • Errors in the SCT.

  • Errors from the learner code.

  • When SCT is waiting on an async event and when its resolved.

Note that in Preview Mode errors in the SCT will cause the submission to fail while otherwise those same errors will pass in Student Mode so that faulty exercises are not impossible for the learner to pass.

Thoroughly test your submission test functions before publishing your course.

Back to top

Sign in to Codecademy