So in this lecture, we're going to discuss what is lexical analysis.
So lexical analysis is the process of turning strings into tokens.
So essentially a token has a type and a value.
So think of things like keywords.
Operators, they are token types.
And then the value would be the plus symbol, the multiplication symbol.
So again, in programming, when we say a key word, what do we mean?
We mean integer, we mean float, we mean double write.
All of those things, those data type names are keywords.
When you when you write them in your code, it's a key word, right?
So this is the job of lexical analysis.
It goes through the source input and it produces token output.
Every token has a type and a value.
And the part of the program that's responsible for lexical analysis is called Alexa OC.
So here's the lexical analysis example we can see to the left we have our source program, which is
a simple main function that sets an integer to 51.
It's passed through lexical analysis.
Through Alexa.
We get the following token output.
We have a keyword with the value in it.
That's our return type.
Yeah, we have an identifier with our functioning.
We have an operator with a left bracket symbol with a right bracket symbol with the left bracket.
We have another keyword and then another identifier, then an operator number and symbol.
And then finally the right bracket.
So what are the benefits of lexical analysis?
Well, we can easily identify source code keywords, symbols and operators.
Right.
So it's less work for the parser because now the pausa has to deal with tokens that it knows the type
and value for, rather than having to parse through the actual source file, which would be a lot more
work because if a parse was passed into the source file, it has a lot more to figure out.
It has to figure out what's the number, what's an operator, all of the work that Alexa already does.
So that's why we create Alexa, right?
So it's less work for the parse up.
Token information is in the right format.
So for example, if we come across a number, say 50, we know that in the source file.
That's a string, right?
It's an ask five and an ask is zero.
Or we take that and we convert it into an actual number stored in memory.
So this is actually stored as an integer.
What once it's gone through lexical analysis.
So that means the puzzle doesn't have to worry about converting this into a number because we already
do it during lack's time.
Lexical analysis time.
So another important thing unnecessary information such as spaces and tabs can be discarded.
We don't have to worry about them.
So.
So there's less work for the parses to deal with.
So here are the token types.
In our compiler we have a token type of identifier.
So this is used for all words with letters, underscores and numbers that are not keywords.
So keywords represent data types.
So you can think like float double integer long.
And control procedure command such as break continue return.
Stuff like that.
That is a keyword in our compiler operators, the kind of self explanatory, you know, five plus ten.
So plus is the operator right in our compiler.
We will also have symbols.
So these are quotes, semicolons, colons and so on.
Numbers are pretty self explanatory.
A numeric number 5090 100 strings have data between quotes.
So whenever we have a string in C, we call it a string, right?
Double quote, HELLOWORLD.
Double quote.
That's a string.
Right.
So same with our compiler.
That's a string.
We're compiling C.
So that's a string.
So that that has a token type of string, anything between these double quotes.
So token type of comments, these are for any comments in our C program, whether they be multi line
comments or normal one line comments, they get created into a token type comment, token and token
type new line that just represents a new line in our program.
So back to the key words.
Here are all of the key words values in our compiler OC unsigned sine char, short integer float, double
long struck union and so on.
Here are all of our operators.
And bear in mind, these operators can be joined together.
So for example, the two right arrows to make a bit shift, right to left arrows to make a bit shift
left.
And then for indirection here pointer access.
Basically we have the asterisks all joined together in this case, a depth of three for a pointer.
Right.
So a number numbers are this algorithm here.
So if the ASCII character we read is above or equal ASCII zero and the character is below our equal
ASCII nine, then it's a number and a number token should be made.
So here are the symbols in our program so we can see left bracket, right bracket, colon, semicolon,
hash tag, backslash, right bracket and an array bracket.
You might be wondering why is the left bracket in the operator and the right bracket in the symbol?
Well, technically, brackets are operators, so that's why the left brackets here.
And then when we find a symbol with the right bracket, we know we're done with that operator.
So that's what that's about.
Know that when it finds an equivalent right bracket symbol, it's done with that operator.
But we'll discuss more in the parser in another lecture.
I just wanted to get that across why we have the left bracket here in the right bracket and symbol,
because I know some of you might be wondering that.
So an identifier is a capital A to a capital Z, a lowercase A to a lowercase Z, followed by 0 to 9
and under scrolls.
So this is the simple rules for variable names, function names and structure names, right?
So anything that is not a keyword and has these rules is an identifier.
So obviously if we type char, it can't be an identifier because that's a known keyword.
So the keyword takes priority.
And then if it's not a keyword, it's an identifier.
So yeah, that that explains lexical analysis in the next few lectures will be actually creating Alexa
following these rules that I've described in this presentation.
Thanks very much for watching and I'll see you in the next lecture.