Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5573 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.6438. Min Act: -0.0002

Data Index: 359587 (C4 (Web Text))

Max Activating Token Index: 830

Text #1

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.6588. Min Act: -0.0002

Data Index: 752632 (C4 (Web Text))

Max Activating Token Index: 717

Text #2

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.5990. Min Act: -0.0002

Data Index: 926614 (C4 (Web Text))

Max Activating Token Index: 152

Text #3

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.5043. Min Act: -0.0002

Data Index: 286499 (C4 (Web Text))

Max Activating Token Index: 805

Text #4

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.5374. Min Act: -0.0002

Data Index: 975750 (C4 (Web Text))

Max Activating Token Index: 393

Text #5

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.5338. Min Act: -0.0002

Data Index: 582993 (C4 (Web Text))

Max Activating Token Index: 492

Text #6

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.4938. Min Act: -0.0002

Data Index: 188677 (C4 (Web Text))

Max Activating Token Index: 614

Text #7

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.4726. Min Act: -0.0002

Data Index: 849264 (C4 (Web Text))

Max Activating Token Index: 701

Text #8

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.4352. Min Act: -0.0002

Data Index: 282354 (C4 (Web Text))

Max Activating Token Index: 655

Text #9

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.4797. Min Act: -0.0002

Data Index: 421985 (C4 (Web Text))

Max Activating Token Index: 500

Text #10

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3887. Min Act: -0.0002

Data Index: 944719 (C4 (Web Text))

Max Activating Token Index: 366

Text #11

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3768. Min Act: -0.0002

Data Index: 1136262 (C4 (Web Text))

Max Activating Token Index: 759

Text #12

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3637. Min Act: -0.0002

Data Index: 473347 (C4 (Web Text))

Max Activating Token Index: 608

Text #13

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3756. Min Act: -0.0002

Data Index: 1257555 (C4 (Web Text))

Max Activating Token Index: 875

Text #14

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3888. Min Act: -0.0002

Data Index: 723447 (C4 (Web Text))

Max Activating Token Index: 585

Text #15

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3738. Min Act: -0.0002

Data Index: 1116855 (C4 (Web Text))

Max Activating Token Index: 540

Text #16

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3223. Min Act: -0.0002

Data Index: 90104 (C4 (Web Text))

Max Activating Token Index: 626

Text #17

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.3202. Min Act: -0.0002

Data Index: 796632 (C4 (Web Text))

Max Activating Token Index: 895

Text #18

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.2844. Min Act: -0.0002

Data Index: 511597 (C4 (Web Text))

Max Activating Token Index: 707

Text #19

Max Range: 1.6588. Min Range: -1.6588

Max Act: 1.2957. Min Act: -0.0002

Data Index: 738825 (C4 (Web Text))

Max Activating Token Index: 24