Operating system security is the set of techniques used to protect a computer system, its users, its programs, and its data. The operating system controls hardware, memory, files, processes, devices, and user accounts, so it is one of the most important layers of security in any computer.
A secure operating system tries to prevent unauthorized access, stop malicious programs, protect sensitive data, isolate users and processes, record suspicious activity, and keep the system available for legitimate users.
The three main goals of security are confidentiality, integrity, and availability. These are known as the CIA Triad.
| Goal | Meaning | OS Example |
|---|---|---|
| Confidentiality | Only authorized users can access information | File permissions, encryption, login protection |
| Integrity | Data and system files are accurate and not tampered with | Checksums, signed updates, access control |
| Availability | Authorized users can access resources when needed | Resource limits, backups, fault tolerance |
An OS faces threats from malware, attackers, unsafe applications, weak passwords, network attacks, and even accidental user mistakes. The table below shows common threats and their impact.
| Threat | Description | Possible Impact |
|---|---|---|
| Virus | Malicious code that attaches itself to another program or file | Data corruption, file infection, system instability |
| Worm | Self-replicating malware that spreads through networks | Network congestion, mass infection |
| Trojan horse | Malware disguised as useful or trusted software | Backdoor access, data theft |
| Ransomware | Encrypts data and demands payment for recovery | Data loss, business interruption |
| Spyware | Secretly monitors user activity | Privacy loss, credential theft |
| Rootkit | Hides malicious activity deep inside the system | Persistent compromise, difficult detection |
| Buffer overflow | Writes more data into memory than a buffer can hold | Crash, code execution, privilege escalation |
| Privilege escalation | An attacker gains permissions beyond what they should have | Full system control |
| Denial of service | Overloads system resources or makes services unavailable | Downtime and service disruption |
Authentication verifies identity. Before the OS grants access, it must know who is trying to log in or run a privileged action.
Multi-factor authentication, or MFA, combines two or more factors. For example, a password plus a one-time code is stronger than a password alone.
| Factor | Meaning | Examples |
|---|---|---|
| Something you know | A secret remembered by the user | Password, PIN, passphrase |
| Something you have | A physical or digital item owned by the user | Smart card, security key, OTP app |
| Something you are | A biometric property | Fingerprint, face recognition, iris scan |
A secure OS should not store plain-text passwords. Passwords should be processed with a secure password hashing algorithm and a unique salt. A salt makes identical passwords produce different stored hashes.
User password + random salt
|
v
Password hashing algorithm
|
v
Stored value: salt + password hash
During login, the OS repeats the hash process and compares hashes.Authentication answers, "Who are you?" Authorization answers, "What are you allowed to do?" The OS uses access control rules to decide whether a user or process can read, write, execute, delete, or modify a resource.
| Model | Full Form | How It Works | Example |
|---|---|---|---|
| DAC | Discretionary Access Control | Resource owner decides who can access the resource | Traditional Unix file permissions |
| MAC | Mandatory Access Control | System policy controls access; users cannot override it | SELinux, AppArmor, military classification systems |
| RBAC | Role-Based Access Control | Permissions are assigned to roles, and users are assigned roles | Admin, manager, developer, guest |
| ABAC | Attribute-Based Access Control | Access depends on user, resource, action, and environment attributes | Cloud IAM policies |
File permissions are one of the most visible parts of OS security. They define who can read, write, or execute a file.
| Permission | Symbol | Meaning for Files | Meaning for Directories |
|---|---|---|---|
| Read | r | View file contents | List directory contents |
| Write | w | Modify file contents | Create, rename, or delete entries inside the directory |
| Execute | x | Run the file as a program or script | Enter or traverse the directory |
-rwxr-x---
| | |
| | +-- others: no permission
| +----- group: read and execute
+-------- owner: read, write, and execute
r = read, w = write, x = executeAn Access Control List, or ACL, defines which users or groups can access a resource and what operations they can perform. ACLs are more flexible than simple owner-group-other permissions.
The principle of least privilege says that every user, program, and process should have only the permissions required to perform its task, and nothing more. This reduces damage if an account or application is compromised.
Modern operating systems separate normal application code from privileged kernel code. This protects the system from accidental or malicious damage by applications.
Applications request privileged services through system calls. This creates a controlled boundary between user programs and the kernel.
| Mode | Access Level | Examples |
|---|---|---|
| User mode | Restricted access to hardware and memory | Browser, editor, media player, user applications |
| Kernel mode | Full privileged access to system resources | Kernel, device drivers, memory manager, scheduler |
Memory attacks are common because many programs store sensitive data and executable instructions in memory. Operating systems and hardware use several protections to reduce memory-based attacks.
| Mechanism | Purpose |
|---|---|
| Process isolation | Prevents one process from directly accessing another process memory |
| Virtual memory | Gives each process its own address space |
| ASLR | Randomizes memory locations to make exploits harder |
| DEP / NX bit | Marks memory regions as non-executable |
| Stack canaries | Detects some stack buffer overflow attacks |
Encryption protects data by converting it into unreadable form without the correct key. Operating systems use encryption to protect stored data, communication, memory, and boot integrity.
| Encryption Type | Description | Examples |
|---|---|---|
| Full disk encryption | Encrypts the entire storage device | BitLocker, FileVault, LUKS |
| File-level encryption | Encrypts selected files or folders | Windows EFS, encrypted home directories |
| Memory encryption | Protects data stored in RAM | AMD SME/SEV, Intel TME |
| Network encryption | Protects data during transmission | TLS, SSH, VPNs |
Boot security protects the startup process. If malware runs before the operating system loads, it can hide from security tools. Secure Boot helps prevent this by verifying digital signatures during startup.
Sandboxing runs an application in a restricted environment. If the application is compromised, the damage is limited because the sandbox blocks access to sensitive files, devices, or system calls.
| Isolation Method | What It Does | Example |
|---|---|---|
| Process sandbox | Restricts a process permissions and system calls | Browser tab sandboxing |
| Container | Isolates applications while sharing the host kernel | Docker, Linux containers |
| Virtual machine | Runs a full guest OS on virtual hardware | VirtualBox, VMware, Hyper-V |
Audit logs record important events such as logins, failed login attempts, file access, privilege changes, service starts, configuration changes, and security policy violations. Logs help administrators detect attacks and investigate incidents.
Many attacks exploit known vulnerabilities. Patch management is the process of applying security updates to the OS, applications, drivers, firmware, and libraries.
Hardening means reducing the attack surface of an operating system. A smaller attack surface gives attackers fewer ways to break in.
The following example shows how multiple OS security controls work together to protect a shared workstation.
Goal: Secure a shared office workstation
1. Create separate user accounts for each employee.
2. Give normal users standard permissions, not administrator rights.
3. Enable full disk encryption.
4. Require strong passwords and MFA where available.
5. Keep automatic security updates enabled.
6. Allow only required applications.
7. Enable firewall protection.
8. Turn on audit logging for login and privilege changes.
9. Back up important files to a protected location.
10. Review accounts and permissions regularly.OS security protects the operating system, users, processes, files, devices, and data from unauthorized access, malware, misuse, and system failure.
The CIA triad stands for confidentiality, integrity, and availability. It is used to describe the main goals of information security.
Authentication verifies identity, while authorization decides what an authenticated user or process is allowed to access.
Least privilege means giving users, processes, and services only the permissions they need to do their job, and no unnecessary access.
Updates fix known vulnerabilities. Without patches, attackers can exploit old bugs that may already have public attack tools.
Explore 500+ free tutorials across 20+ languages and frameworks.