Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2032 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5978. Min Act: -0.0001

Data Index: 1265441 (C4 (Web Text))

Max Activating Token Index: 431

Text #1

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5954. Min Act: -0.0001

Data Index: 1081975 (C4 (Web Text))

Max Activating Token Index: 120

Text #2

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.6068. Min Act: -0.0001

Data Index: 811097 (C4 (Web Text))

Max Activating Token Index: 865

Text #3

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5918. Min Act: -0.0001

Data Index: 1337355 (C4 (Web Text))

Max Activating Token Index: 741

Text #4

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5804. Min Act: -0.0001

Data Index: 83460 (C4 (Web Text))

Max Activating Token Index: 792

Text #5

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5332. Min Act: -0.0001

Data Index: 45537 (C4 (Web Text))

Max Activating Token Index: 493

Text #6

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5233. Min Act: -0.0001

Data Index: 1217246 (C4 (Web Text))

Max Activating Token Index: 647

Text #7

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5448. Min Act: -0.0001

Data Index: 62587 (C4 (Web Text))

Max Activating Token Index: 27

Text #8

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5409. Min Act: -0.0001

Data Index: 1252773 (C4 (Web Text))

Max Activating Token Index: 819

Text #9

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5282. Min Act: -0.0001

Data Index: 382704 (C4 (Web Text))

Max Activating Token Index: 257

Text #10

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5431. Min Act: -0.0001

Data Index: 1111094 (C4 (Web Text))

Max Activating Token Index: 951

Text #11

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5202. Min Act: -0.0001

Data Index: 336850 (C4 (Web Text))

Max Activating Token Index: 762

Text #12

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5032. Min Act: -0.0001

Data Index: 694502 (C4 (Web Text))

Max Activating Token Index: 465

Text #13

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.5086. Min Act: -0.0001

Data Index: 779350 (C4 (Web Text))

Max Activating Token Index: 746

Text #14

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.4851. Min Act: -0.0001

Data Index: 525622 (C4 (Web Text))

Max Activating Token Index: 159

Text #15

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.4789. Min Act: -0.0001

Data Index: 965537 (C4 (Web Text))

Max Activating Token Index: 888

Text #16

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.4945. Min Act: -0.0001

Data Index: 1115255 (C4 (Web Text))

Max Activating Token Index: 333

Text #17

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.4858. Min Act: -0.0001

Data Index: 1111191 (C4 (Web Text))

Max Activating Token Index: 679

Text #18

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.4838. Min Act: -0.0001

Data Index: 101702 (C4 (Web Text))

Max Activating Token Index: 55

Text #19

Max Range: 0.6068. Min Range: -0.6068

Max Act: 0.4970. Min Act: -0.0001

Data Index: 926131 (C4 (Web Text))

Max Activating Token Index: 423