Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1183 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.6074. Min Act: -0.0000

Data Index: 119023 (C4 (Web Text))

Max Activating Token Index: 699

Text #1

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.6313. Min Act: -0.0000

Data Index: 946373 (C4 (Web Text))

Max Activating Token Index: 904

Text #2

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.6285. Min Act: 0.0000

Data Index: 1464275 (Python Code)

Max Activating Token Index: 368

Text #3

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.6136. Min Act: 0.0002

Data Index: 340938 (C4 (Web Text))

Max Activating Token Index: 317

Text #4

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5979. Min Act: -0.0000

Data Index: 782379 (C4 (Web Text))

Max Activating Token Index: 389

Text #5

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5962. Min Act: -0.0000

Data Index: 20915 (C4 (Web Text))

Max Activating Token Index: 102

Text #6

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5759. Min Act: 0.0002

Data Index: 1492578 (Python Code)

Max Activating Token Index: 29

Text #7

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5565. Min Act: -0.0000

Data Index: 1329590 (C4 (Web Text))

Max Activating Token Index: 419

Text #8

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5678. Min Act: -0.0000

Data Index: 323956 (C4 (Web Text))

Max Activating Token Index: 529

Text #9

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5637. Min Act: 0.0001

Data Index: 475380 (C4 (Web Text))

Max Activating Token Index: 958

Text #10

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5374. Min Act: -0.0000

Data Index: 426101 (C4 (Web Text))

Max Activating Token Index: 76

Text #11

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5696. Min Act: -0.0000

Data Index: 1609473 (Python Code)

Max Activating Token Index: 261

Text #12

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5323. Min Act: -0.0000

Data Index: 1217321 (C4 (Web Text))

Max Activating Token Index: 1003

Text #13

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5398. Min Act: -0.0000

Data Index: 1314006 (C4 (Web Text))

Max Activating Token Index: 1020

Text #14

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5160. Min Act: 0.0003

Data Index: 491411 (C4 (Web Text))

Max Activating Token Index: 116

Text #15

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5117. Min Act: -0.0000

Data Index: 933399 (C4 (Web Text))

Max Activating Token Index: 720

Text #16

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5271. Min Act: -0.0000

Data Index: 473053 (C4 (Web Text))

Max Activating Token Index: 700

Text #17

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.5083. Min Act: -0.0000

Data Index: 1402563 (Python Code)

Max Activating Token Index: 267

Text #18

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.4929. Min Act: -0.0000

Data Index: 586537 (C4 (Web Text))

Max Activating Token Index: 98

Text #19

Max Range: 0.6313. Min Range: -0.6313

Max Act: 0.4882. Min Act: -0.0000

Data Index: 577489 (C4 (Web Text))

Max Activating Token Index: 963