Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2207 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.4203. Min Act: -0.0001

Data Index: 471926 (C4 (Web Text))

Max Activating Token Index: 921

Text #1

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.4233. Min Act: -0.0001

Data Index: 738509 (C4 (Web Text))

Max Activating Token Index: 874

Text #2

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.4099. Min Act: -0.0001

Data Index: 230669 (C4 (Web Text))

Max Activating Token Index: 847

Text #3

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.4071. Min Act: -0.0001

Data Index: 461600 (C4 (Web Text))

Max Activating Token Index: 947

Text #4

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3542. Min Act: -0.0001

Data Index: 1222617 (C4 (Web Text))

Max Activating Token Index: 852

Text #5

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3658. Min Act: -0.0001

Data Index: 916611 (C4 (Web Text))

Max Activating Token Index: 733

Text #6

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3445. Min Act: -0.0001

Data Index: 831091 (C4 (Web Text))

Max Activating Token Index: 995

Text #7

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3519. Min Act: -0.0001

Data Index: 426222 (C4 (Web Text))

Max Activating Token Index: 1016

Text #8

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3279. Min Act: -0.0001

Data Index: 24395 (C4 (Web Text))

Max Activating Token Index: 928

Text #9

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3544. Min Act: -0.0001

Data Index: 8352 (C4 (Web Text))

Max Activating Token Index: 921

Text #10

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3474. Min Act: -0.0001

Data Index: 399505 (C4 (Web Text))

Max Activating Token Index: 836

Text #11

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3351. Min Act: -0.0001

Data Index: 632871 (C4 (Web Text))

Max Activating Token Index: 1021

Text #12

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3300. Min Act: -0.0001

Data Index: 1163882 (C4 (Web Text))

Max Activating Token Index: 854

Text #13

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3281. Min Act: -0.0001

Data Index: 654832 (C4 (Web Text))

Max Activating Token Index: 448

Text #14

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3387. Min Act: -0.0001

Data Index: 733998 (C4 (Web Text))

Max Activating Token Index: 914

Text #15

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3262. Min Act: -0.0001

Data Index: 155110 (C4 (Web Text))

Max Activating Token Index: 36

Text #16

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3256. Min Act: -0.0001

Data Index: 174753 (C4 (Web Text))

Max Activating Token Index: 31

Text #17

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3246. Min Act: -0.0001

Data Index: 444023 (C4 (Web Text))

Max Activating Token Index: 527

Text #18

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3142. Min Act: -0.0001

Data Index: 1122078 (C4 (Web Text))

Max Activating Token Index: 900

Text #19

Max Range: 0.4233. Min Range: -0.4233

Max Act: 0.3143. Min Act: -0.0001

Data Index: 1122238 (C4 (Web Text))

Max Activating Token Index: 888