Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4595 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.2596. Min Act: -0.0000

Data Index: 1209973 (C4 (Web Text))

Max Activating Token Index: 809

Text #1

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.2014. Min Act: -0.0000

Data Index: 168274 (C4 (Web Text))

Max Activating Token Index: 242

Text #2

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1837. Min Act: -0.0000

Data Index: 1251754 (C4 (Web Text))

Max Activating Token Index: 191

Text #3

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1849. Min Act: -0.0000

Data Index: 603932 (C4 (Web Text))

Max Activating Token Index: 236

Text #4

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1260. Min Act: -0.0000

Data Index: 1208263 (C4 (Web Text))

Max Activating Token Index: 377

Text #5

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.2775. Min Act: -0.0000

Data Index: 1242667 (C4 (Web Text))

Max Activating Token Index: 560

Text #6

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1647. Min Act: -0.0000

Data Index: 401608 (C4 (Web Text))

Max Activating Token Index: 614

Text #7

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1373. Min Act: -0.0000

Data Index: 527215 (C4 (Web Text))

Max Activating Token Index: 699

Text #8

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.0964. Min Act: -0.0000

Data Index: 1064454 (C4 (Web Text))

Max Activating Token Index: 61

Text #9

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.3126. Min Act: -0.0000

Data Index: 1150821 (C4 (Web Text))

Max Activating Token Index: 185

Text #10

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.0685. Min Act: -0.0000

Data Index: 1312507 (C4 (Web Text))

Max Activating Token Index: 102

Text #11

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1291. Min Act: -0.0000

Data Index: 1176108 (C4 (Web Text))

Max Activating Token Index: 236

Text #12

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1897. Min Act: -0.0000

Data Index: 24139 (C4 (Web Text))

Max Activating Token Index: 243

Text #13

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1753. Min Act: -0.0000

Data Index: 429675 (C4 (Web Text))

Max Activating Token Index: 320

Text #14

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1832. Min Act: -0.0000

Data Index: 1150059 (C4 (Web Text))

Max Activating Token Index: 389

Text #15

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1912. Min Act: -0.0000

Data Index: 503478 (C4 (Web Text))

Max Activating Token Index: 308

Text #16

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1274. Min Act: -0.0000

Data Index: 1144789 (C4 (Web Text))

Max Activating Token Index: 395

Text #17

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.1784. Min Act: -0.0000

Data Index: 868434 (C4 (Web Text))

Max Activating Token Index: 124

Text #18

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.2947. Min Act: -0.0000

Data Index: 721651 (C4 (Web Text))

Max Activating Token Index: 432

Text #19

Max Range: 2.3126. Min Range: -2.3126

Max Act: 2.0721. Min Act: -0.0000

Data Index: 1006509 (C4 (Web Text))

Max Activating Token Index: 349