OS Security — Authentication, DAC, MAC, RBAC

What is OS Security?

Operating system security is the set of techniques used to protect a computer system, its users, its programs, and its data. The operating system controls hardware, memory, files, processes, devices, and user accounts, so it is one of the most important layers of security in any computer.

A secure operating system tries to prevent unauthorized access, stop malicious programs, protect sensitive data, isolate users and processes, record suspicious activity, and keep the system available for legitimate users.

Security Goals: CIA Triad

The three main goals of security are confidentiality, integrity, and availability. These are known as the CIA Triad.

Additional Security Goals

• Authentication: Verifies who the user or process is.
• Authorization: Decides what an authenticated user or process can do.
• Accountability: Records actions so they can be traced later.
• Non-repudiation: Prevents a user from denying an action they performed.
• Isolation: Keeps processes, users, and resources separated from each other.

Security Threats in Operating Systems

An OS faces threats from malware, attackers, unsafe applications, weak passwords, network attacks, and even accidental user mistakes. The table below shows common threats and their impact.

Authentication

Authentication verifies identity. Before the OS grants access, it must know who is trying to log in or run a privileged action.

Multi-factor authentication, or MFA, combines two or more factors. For example, a password plus a one-time code is stronger than a password alone.

Password Storage and Protection

A secure OS should not store plain-text passwords. Passwords should be processed with a secure password hashing algorithm and a unique salt. A salt makes identical passwords produce different stored hashes.

• Store password hashes, not plain passwords.
• Use slow password hashing algorithms such as bcrypt, scrypt, or Argon2.
• Use account lockout or rate limiting to slow brute-force attempts.
• Prefer long passphrases and MFA for important accounts.

User password + random salt
        |
        v
Password hashing algorithm
        |
        v
Stored value: salt + password hash

During login, the OS repeats the hash process and compares hashes.

Authorization and Access Control

Authentication answers, "Who are you?" Authorization answers, "What are you allowed to do?" The OS uses access control rules to decide whether a user or process can read, write, execute, delete, or modify a resource.

File Permissions

File permissions are one of the most visible parts of OS security. They define who can read, write, or execute a file.

-rwxr-x---
 |  |  |
 |  |  +-- others: no permission
 |  +----- group: read and execute
 +-------- owner: read, write, and execute

r = read, w = write, x = execute

Access Control Lists (ACLs)

An Access Control List, or ACL, defines which users or groups can access a resource and what operations they can perform. ACLs are more flexible than simple owner-group-other permissions.

• A file ACL can give Alice read/write access and Bob read-only access.
• A network ACL can allow or block traffic by IP address, port, or protocol.
• A capability list is the reverse idea: it lists what resources a user can access.

Principle of Least Privilege

The principle of least privilege says that every user, program, and process should have only the permissions required to perform its task, and nothing more. This reduces damage if an account or application is compromised.

• Use normal user accounts for daily work instead of administrator accounts.
• Give services only the file and network access they actually need.
• Remove unused permissions, users, groups, and services.
• Use privilege elevation only when necessary.

User Mode and Kernel Mode

Modern operating systems separate normal application code from privileged kernel code. This protects the system from accidental or malicious damage by applications.

Applications request privileged services through system calls. This creates a controlled boundary between user programs and the kernel.

Memory Protection Mechanisms

Memory attacks are common because many programs store sensitive data and executable instructions in memory. Operating systems and hardware use several protections to reduce memory-based attacks.

Encryption in OS Security

Encryption protects data by converting it into unreadable form without the correct key. Operating systems use encryption to protect stored data, communication, memory, and boot integrity.

Secure Boot and Trusted Boot

Boot security protects the startup process. If malware runs before the operating system loads, it can hide from security tools. Secure Boot helps prevent this by verifying digital signatures during startup.

• Secure Boot: Allows only trusted bootloaders and OS components to load.
• Trusted Platform Module (TPM): Stores cryptographic measurements and keys securely.
• Measured boot: Records boot component measurements for later verification.
• Code signing: Confirms that software comes from a trusted publisher and has not been changed.

Sandboxing and Isolation

Sandboxing runs an application in a restricted environment. If the application is compromised, the damage is limited because the sandbox blocks access to sensitive files, devices, or system calls.

Auditing and Logging

Audit logs record important events such as logins, failed login attempts, file access, privilege changes, service starts, configuration changes, and security policy violations. Logs help administrators detect attacks and investigate incidents.

• Track successful and failed authentication attempts.
• Record privilege elevation and administrator actions.
• Monitor important file and configuration changes.
• Send logs to a protected central location when possible.

Patch Management

Many attacks exploit known vulnerabilities. Patch management is the process of applying security updates to the OS, applications, drivers, firmware, and libraries.

1. Identify available security updates.
2. Test updates when the system is critical.
3. Apply patches regularly.
4. Reboot when required.
5. Verify that the patch was installed successfully.

OS Hardening Checklist

Hardening means reducing the attack surface of an operating system. A smaller attack surface gives attackers fewer ways to break in.

• Install security updates regularly.
• Disable unused services and startup programs.
• Use strong authentication and MFA for sensitive accounts.
• Apply least privilege to users, services, and files.
• Enable firewall rules and allow only required traffic.
• Use disk encryption on laptops and portable devices.
• Enable logging and review important security events.
• Keep backups and test restore procedures.

Complete Security Example

The following example shows how multiple OS security controls work together to protect a shared workstation.

Goal: Secure a shared office workstation

1. Create separate user accounts for each employee.
2. Give normal users standard permissions, not administrator rights.
3. Enable full disk encryption.
4. Require strong passwords and MFA where available.
5. Keep automatic security updates enabled.
6. Allow only required applications.
7. Enable firewall protection.
8. Turn on audit logging for login and privilege changes.
9. Back up important files to a protected location.
10. Review accounts and permissions regularly.

Common Security Mistakes

• Using administrator accounts for everyday work.
• Reusing weak passwords across many systems.
• Ignoring OS and application updates.
• Giving broad file permissions such as full access to everyone.
• Running unknown software from untrusted sources.
• Keeping unused services open on the network.
• Having backups but never testing restore procedures.