Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1450 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.5788. Min Act: -0.0001

Data Index: 888790 (C4 (Web Text))

Max Activating Token Index: 592

Text #1

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.4947. Min Act: -0.0001

Data Index: 348599 (C4 (Web Text))

Max Activating Token Index: 634

Text #2

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.4202. Min Act: -0.0001

Data Index: 943762 (C4 (Web Text))

Max Activating Token Index: 697

Text #3

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.4476. Min Act: -0.0001

Data Index: 1221693 (C4 (Web Text))

Max Activating Token Index: 693

Text #4

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.4814. Min Act: -0.0001

Data Index: 998001 (C4 (Web Text))

Max Activating Token Index: 354

Text #5

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3599. Min Act: -0.0001

Data Index: 355492 (C4 (Web Text))

Max Activating Token Index: 533

Text #6

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3035. Min Act: -0.0001

Data Index: 438743 (C4 (Web Text))

Max Activating Token Index: 913

Text #7

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3732. Min Act: -0.0001

Data Index: 97542 (C4 (Web Text))

Max Activating Token Index: 497

Text #8

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2753. Min Act: -0.0001

Data Index: 1331240 (C4 (Web Text))

Max Activating Token Index: 256

Text #9

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3565. Min Act: -0.0001

Data Index: 125049 (C4 (Web Text))

Max Activating Token Index: 671

Text #10

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2789. Min Act: -0.0001

Data Index: 908607 (C4 (Web Text))

Max Activating Token Index: 489

Text #11

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2591. Min Act: -0.0001

Data Index: 1274898 (C4 (Web Text))

Max Activating Token Index: 350

Text #12

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2996. Min Act: -0.0001

Data Index: 845129 (C4 (Web Text))

Max Activating Token Index: 466

Text #13

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2910. Min Act: -0.0001

Data Index: 386189 (C4 (Web Text))

Max Activating Token Index: 218

Text #14

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2529. Min Act: -0.0001

Data Index: 447386 (C4 (Web Text))

Max Activating Token Index: 628

Text #15

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3368. Min Act: -0.0001

Data Index: 1016532 (C4 (Web Text))

Max Activating Token Index: 598

Text #16

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3352. Min Act: -0.0001

Data Index: 682094 (C4 (Web Text))

Max Activating Token Index: 404

Text #17

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2873. Min Act: -0.0001

Data Index: 585563 (C4 (Web Text))

Max Activating Token Index: 449

Text #18

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.3700. Min Act: -0.0001

Data Index: 444296 (C4 (Web Text))

Max Activating Token Index: 609

Text #19

Max Range: 1.5788. Min Range: -1.5788

Max Act: 1.2496. Min Act: -0.0001

Data Index: 1020272 (C4 (Web Text))

Max Activating Token Index: 688