Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 440 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0785. Min Act: -0.0002

Data Index: 306447 (C4 (Web Text))

Max Activating Token Index: 678

Text #1

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0706. Min Act: -0.0002

Data Index: 118037 (C4 (Web Text))

Max Activating Token Index: 27

Text #2

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0687. Min Act: -0.0002

Data Index: 662938 (C4 (Web Text))

Max Activating Token Index: 156

Text #3

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0682. Min Act: -0.0002

Data Index: 1014342 (C4 (Web Text))

Max Activating Token Index: 36

Text #4

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0678. Min Act: -0.0002

Data Index: 631570 (C4 (Web Text))

Max Activating Token Index: 352

Text #5

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0661. Min Act: -0.0002

Data Index: 1353160 (C4 (Web Text))

Max Activating Token Index: 767

Text #6

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0657. Min Act: -0.0002

Data Index: 311409 (C4 (Web Text))

Max Activating Token Index: 342

Text #7

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0662. Min Act: -0.0002

Data Index: 428740 (C4 (Web Text))

Max Activating Token Index: 221

Text #8

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0676. Min Act: -0.0002

Data Index: 350913 (C4 (Web Text))

Max Activating Token Index: 499

Text #9

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0631. Min Act: -0.0002

Data Index: 1349843 (C4 (Web Text))

Max Activating Token Index: 975

Text #10

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0632. Min Act: -0.0002

Data Index: 1030009 (C4 (Web Text))

Max Activating Token Index: 418

Text #11

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0609. Min Act: -0.0002

Data Index: 156413 (C4 (Web Text))

Max Activating Token Index: 852

Text #12

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0635. Min Act: -0.0002

Data Index: 1087951 (C4 (Web Text))

Max Activating Token Index: 608

Text #13

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0616. Min Act: -0.0002

Data Index: 1291133 (C4 (Web Text))

Max Activating Token Index: 856

Text #14

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0620. Min Act: -0.0002

Data Index: 57208 (C4 (Web Text))

Max Activating Token Index: 646

Text #15

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0618. Min Act: -0.0002

Data Index: 754480 (C4 (Web Text))

Max Activating Token Index: 856

Text #16

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0618. Min Act: -0.0002

Data Index: 381841 (C4 (Web Text))

Max Activating Token Index: 143

Text #17

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0606. Min Act: -0.0002

Data Index: 143067 (C4 (Web Text))

Max Activating Token Index: 33

Text #18

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0626. Min Act: -0.0002

Data Index: 1250283 (C4 (Web Text))

Max Activating Token Index: 827

Text #19

Max Range: 0.0785. Min Range: -0.0785

Max Act: 0.0621. Min Act: -0.0002

Data Index: 1043088 (C4 (Web Text))

Max Activating Token Index: 534