Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3456 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.6500. Min Range: -6.6500

Max Act: 6.5274. Min Act: -0.0000

Data Index: 1321700 (C4 (Web Text))

Max Activating Token Index: 227

Text #1

Max Range: 6.6500. Min Range: -6.6500

Max Act: 6.6500. Min Act: -0.0000

Data Index: 830703 (C4 (Web Text))

Max Activating Token Index: 56

Text #2

Max Range: 6.6500. Min Range: -6.6500

Max Act: 6.0742. Min Act: -0.0000

Data Index: 366991 (C4 (Web Text))

Max Activating Token Index: 14

Text #3

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.9988. Min Act: -0.0000

Data Index: 761953 (C4 (Web Text))

Max Activating Token Index: 221

Text #4

Max Range: 6.6500. Min Range: -6.6500

Max Act: 6.1828. Min Act: -0.0000

Data Index: 1111983 (C4 (Web Text))

Max Activating Token Index: 39

Text #5

Max Range: 6.6500. Min Range: -6.6500

Max Act: 6.2041. Min Act: -0.0000

Data Index: 579208 (C4 (Web Text))

Max Activating Token Index: 191

Text #6

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.7477. Min Act: -0.0000

Data Index: 723556 (C4 (Web Text))

Max Activating Token Index: 77

Text #7

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.6673. Min Act: -0.0000

Data Index: 165431 (C4 (Web Text))

Max Activating Token Index: 1014

Text #8

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.5925. Min Act: -0.0000

Data Index: 1028324 (C4 (Web Text))

Max Activating Token Index: 30

Text #9

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.5058. Min Act: -0.0000

Data Index: 1176293 (C4 (Web Text))

Max Activating Token Index: 132

Text #10

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.4621. Min Act: -0.0000

Data Index: 67527 (C4 (Web Text))

Max Activating Token Index: 989

Text #11

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.4270. Min Act: -0.0000

Data Index: 255953 (C4 (Web Text))

Max Activating Token Index: 589

Text #12

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.2162. Min Act: -0.0000

Data Index: 692273 (C4 (Web Text))

Max Activating Token Index: 835

Text #13

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.3997. Min Act: -0.0000

Data Index: 312393 (C4 (Web Text))

Max Activating Token Index: 677

Text #14

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.1828. Min Act: -0.0000

Data Index: 2608 (C4 (Web Text))

Max Activating Token Index: 665

Text #15

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.2741. Min Act: -0.0000

Data Index: 837423 (C4 (Web Text))

Max Activating Token Index: 1014

Text #16

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.4663. Min Act: -0.0000

Data Index: 1099718 (C4 (Web Text))

Max Activating Token Index: 442

Text #17

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.6330. Min Act: -0.0000

Data Index: 995171 (C4 (Web Text))

Max Activating Token Index: 737

Text #18

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.2518. Min Act: -0.0000

Data Index: 1214146 (C4 (Web Text))

Max Activating Token Index: 20

Text #19

Max Range: 6.6500. Min Range: -6.6500

Max Act: 5.2367. Min Act: -0.0000

Data Index: 535368 (C4 (Web Text))

Max Activating Token Index: 682