Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1935 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2829. Min Act: -0.0002

Data Index: 503382 (C4 (Web Text))

Max Activating Token Index: 267

Text #1

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2705. Min Act: -0.0002

Data Index: 830033 (C4 (Web Text))

Max Activating Token Index: 104

Text #2

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2205. Min Act: -0.0002

Data Index: 1113869 (C4 (Web Text))

Max Activating Token Index: 471

Text #3

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2056. Min Act: -0.0003

Data Index: 1199914 (C4 (Web Text))

Max Activating Token Index: 243

Text #4

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2127. Min Act: -0.0002

Data Index: 714320 (C4 (Web Text))

Max Activating Token Index: 691

Text #5

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2045. Min Act: -0.0002

Data Index: 207874 (C4 (Web Text))

Max Activating Token Index: 233

Text #6

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2029. Min Act: -0.0002

Data Index: 628247 (C4 (Web Text))

Max Activating Token Index: 753

Text #7

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.2059. Min Act: -0.0002

Data Index: 773449 (C4 (Web Text))

Max Activating Token Index: 304

Text #8

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1921. Min Act: -0.0002

Data Index: 939444 (C4 (Web Text))

Max Activating Token Index: 450

Text #9

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1935. Min Act: -0.0002

Data Index: 419674 (C4 (Web Text))

Max Activating Token Index: 530

Text #10

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1894. Min Act: -0.0002

Data Index: 474050 (C4 (Web Text))

Max Activating Token Index: 74

Text #11

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1897. Min Act: -0.0002

Data Index: 490761 (C4 (Web Text))

Max Activating Token Index: 945

Text #12

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1828. Min Act: -0.0002

Data Index: 1001244 (C4 (Web Text))

Max Activating Token Index: 194

Text #13

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1791. Min Act: -0.0002

Data Index: 604003 (C4 (Web Text))

Max Activating Token Index: 423

Text #14

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1850. Min Act: -0.0002

Data Index: 592960 (C4 (Web Text))

Max Activating Token Index: 506

Text #15

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1877. Min Act: -0.0002

Data Index: 1034823 (C4 (Web Text))

Max Activating Token Index: 96

Text #16

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1909. Min Act: -0.0002

Data Index: 375724 (C4 (Web Text))

Max Activating Token Index: 994

Text #17

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1844. Min Act: -0.0002

Data Index: 206756 (C4 (Web Text))

Max Activating Token Index: 491

Text #18

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1848. Min Act: -0.0002

Data Index: 1256909 (C4 (Web Text))

Max Activating Token Index: 1022

Text #19

Max Range: 0.2829. Min Range: -0.2829

Max Act: 0.1853. Min Act: -0.0002

Data Index: 191955 (C4 (Web Text))

Max Activating Token Index: 224