Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 804 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2375. Min Act: -0.0000

Data Index: 902765 (C4 (Web Text))

Max Activating Token Index: 344

Text #1

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2306. Min Act: -0.0000

Data Index: 396763 (C4 (Web Text))

Max Activating Token Index: 963

Text #2

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2311. Min Act: -0.0000

Data Index: 655912 (C4 (Web Text))

Max Activating Token Index: 311

Text #3

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2356. Min Act: -0.0000

Data Index: 1171750 (C4 (Web Text))

Max Activating Token Index: 304

Text #4

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2271. Min Act: -0.0000

Data Index: 1356516 (C4 (Web Text))

Max Activating Token Index: 326

Text #5

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2293. Min Act: -0.0000

Data Index: 1062353 (C4 (Web Text))

Max Activating Token Index: 666

Text #6

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2238. Min Act: -0.0000

Data Index: 372800 (C4 (Web Text))

Max Activating Token Index: 290

Text #7

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2242. Min Act: -0.0000

Data Index: 716483 (C4 (Web Text))

Max Activating Token Index: 447

Text #8

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2195. Min Act: -0.0000

Data Index: 660359 (C4 (Web Text))

Max Activating Token Index: 209

Text #9

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2012. Min Act: -0.0000

Data Index: 1291107 (C4 (Web Text))

Max Activating Token Index: 690

Text #10

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2144. Min Act: -0.0000

Data Index: 1048843 (C4 (Web Text))

Max Activating Token Index: 472

Text #11

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2164. Min Act: -0.0000

Data Index: 764291 (C4 (Web Text))

Max Activating Token Index: 255

Text #12

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2154. Min Act: -0.0000

Data Index: 1306886 (C4 (Web Text))

Max Activating Token Index: 88

Text #13

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2083. Min Act: -0.0000

Data Index: 312693 (C4 (Web Text))

Max Activating Token Index: 538

Text #14

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2011. Min Act: -0.0000

Data Index: 1310442 (C4 (Web Text))

Max Activating Token Index: 186

Text #15

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2037. Min Act: -0.0000

Data Index: 93958 (C4 (Web Text))

Max Activating Token Index: 410

Text #16

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2040. Min Act: -0.0000

Data Index: 256271 (C4 (Web Text))

Max Activating Token Index: 414

Text #17

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.2030. Min Act: -0.0000

Data Index: 484986 (C4 (Web Text))

Max Activating Token Index: 506

Text #18

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.1955. Min Act: -0.0000

Data Index: 1011918 (C4 (Web Text))

Max Activating Token Index: 251

Text #19

Max Range: 0.2375. Min Range: -0.2375

Max Act: 0.1939. Min Act: -0.0000

Data Index: 1069375 (C4 (Web Text))

Max Activating Token Index: 441